Skip to content

fix(grader): match repeated tools one-to-one - #235

Merged
helloml0326 merged 1 commit into
agentscope-ai:mainfrom
anticipate218:fix/one-to-one-tool-step-matching
Nov 13, 2026
Merged

helloml0326 merged 1 commit into
agentscope-ai:mainfrom
anticipate218:fix/one-to-one-tool-step-matching

Conversation

@anticipate218

Copy link
Copy Markdown
Contributor

OpenJudge Version

0.2.5, based on 1925e6d9f0ee3467dec07e867b26cec62a708145.

Description

Strict step-by-step scoring (strict_mode=True, use_jaccard_similarity=False) reuses the first prediction with a matching tool name for every reference call. With two calls search(A), search(B) on both sides, it returns 0.75 instead of 1.0. Conversely, predictions search(A), search(B) against references search(A), search(A) receive 1.0 because the first prediction is counted twice.

Group calls by name and maximize total parameter F1 with a one-to-one assignment within each group. This keeps matching independent of order and avoids the suboptimal assignments possible with a greedy algorithm. The existing tool-count requirement, step boundaries, loose mode and Jaccard mode are preserved. SciPy is declared as a direct dependency (>=1.8.0) for linear_sum_assignment; it was already installed transitively through the mandatory scikit-learn dependency.

Twelve regression cases cover exact/reordered matches, repeated predictions, unequal counts, a non-greedy optimum, and separation across tool names and steps. Documentation describes the matching behavior.

Validation

  • Original implementation with the final regression file: 6 failed, 19 passed; fixed implementation: 25 passed.
  • Real installed package: changed test file 25 passed; complete agent/tool tests 81 passed, 11 skipped.
  • Repository pre-commit test scope: 1205 passed, 143 skipped, 1 failed. Original source in the same environment: 1193 passed, 143 skipped, the same 1 failure. The failure is test_installed_codex_accepts_command_arguments_without_model_call: this Windows environment finds codex.CMD, but the existing subprocess invocation raises FileNotFoundError. The unchanged NumPy constant-correlation tests emit five warnings on both versions.
  • Independent one-to-one assignment comparison: 1058 cases matched an exhaustive oracle; 1200 existing-mode controls were unchanged. A separate 700-case randomized exhaustive oracle also passed.
  • Configured Black 25.9.0, isort 5.13.2, Flake8 7.3.0, Pylint 4.0.2 and applicable file hooks passed for changed files. Pyroma 5.0 scored 10/10. pip check passed.

Validation used Windows CPython 3.13.5 and the project's declared editable dev dependencies (including SciPy 1.18.1). The complete pre-commit wrapper, other platforms/Python versions, minimum dependency versions, live API quality/integration tests, multimodal and generator test suites were not run. No model calls were made. Two unrelated documentation/UI image assets were unavailable in the materialized source; all Python source and test data were present. The full test command used the exclusions configured by the repository's pre-commit hook.

AI assistance

Implemented and checked with OpenAI Codex, including AI-assisted code, tests, and independent review, under the account owner's authorization.

Checklist

  • Code has been formatted with pre-commit run --all-files command
  • All tests are passing
  • Docstrings are in Google style
  • Related documentation has been updated (e.g. links, examples, etc.)
  • Code is ready for review

The first two boxes are deliberately unchecked: the configured checks were run directly, and the existing Windows CLI test failure is disclosed above.

Match same-name tool calls using a maximum-weight one-to-one
assignment. Keep step, count and legacy mode behavior unchanged.

Assisted-by: OpenAI Codex
@helloml0326
helloml0326 merged commit e0693fc into agentscope-ai:main Nov 13, 2026
1 check passed
@anticipate218

Copy link
Copy Markdown
Contributor Author

Thanks for reviewing and merging this! I appreciate your time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants